I have huge respect for the apostle Paul, but I think that he lost the plot when confronted by James during his visit to Jerusalem bearing a gift from the Gentile churches (Acts 21). The churches he had planted were coming under intense pressure from Judaizers, who were trying to force Gentile Christians to get circumcised and eat separately from non-Christians.

According to the letter to Galatians, James seems to have been encouraging them (Gal 2:12-13). These Jewish Christians seemed to be defying the agreement made at Antioch (Acts 15), which was not a good agreement anyway. They were going out to all the churches that Paul had planted and trying to bring them back under their way of thinking.

Paul went to Jerusalem in one last desperate attempt to sort out the problem, before he set out on another mission in the direction of Spain. He was bringing a generous gift that was his fulfilment of the agreement he had made with Peter and James during the meeting at Antioch, an agreement that James was not really honouring.

Paul was under intense pressure. He feared for the future of the Gentile mission to which he had devoted his life. The intensity of his feeling is evident in his letters to Galatians and the Philippians. Paul realised that the greatest growth of the gospel would come through the Gentile church, so he didn't want anything to prevent it. His fears were exacerbated by the prophecies he had received on his way warning that he would get a negative reaction in Jerusalem.

I sense that James set a bit of a trap for Paul when he suggested that he pay for the four men taking Nazarite vows. There are hints that the Church in Jerusalem had refused to accept the generous financial gift that Paul brought because they considered it was contaminated by coming from Gentile Christians. As a compromise, James suggested that Paul use some of it to support the four men taking a Nazarite, presumably as a way of legitimising the gift.

Paul had reported the wonderful works that God had done amongst the gentiles, but James seemed to have fallen foul of the fear of men because he claimed that the Jewish Christians were upset by Paul and seemed to be worried that they would stir up trouble. A riot would make life difficult for the apostles who had chosen to stay in Jerusalem, despite Jesus' command to go.

Many thousands of Jews have believed, and all of them are zealous for the law. They have been informed that you teach all the Jews who live among the Gentiles to turn away from Moses, telling them not to circumcise their children or live according to our customs (Acts 19:20-21).
What James claimed they were saying was only half true. Paul had not turned away from Moses, but he did teach that the Gentiles did not need to be circumcised and that they did not need to comply with Jewish customs, such as eating separately from Gentiles.

James already knew what Paul was teaching, and he knew that God was blessing it. James had approved the gospel that Paul preached on a couple of occasions (Gal 1:18-19; 15:13-19), so instead of hitting up Paul, he should have been speaking to the Judaizing group and correcting their misunderstanding of the gospel. James should have been explaining that Gentiles did not need to get circumcised and did not need to comply with Jewish food customs. Instead, he had allowed this problem to fester amongst the Jewish Christians. Then when Paul arrived, he expected him to solve the problem that this created.

Asking Paul to fund the four men taking a Nazarite vow only added to the confusion about the Torah. Jesus had explained that swearing vows was unnecessary for those who had chosen to follow him and receive the gift of the Spirit (Matt 5:34-37), so taking a Nazarite vow was pointless. Once Jesus had died on the cross, the temple sacrifices had become redundant, so the sacrifices Paul was paying for had no value and were a waste of the gift given to support the church in Jerusalem. James was requiring Paul to support Jewish practices from the Torah that had been fulfilled by Jesus and were no longer mandatory for his followers.

Getting Paul to be involved in redundant Torah practices did not help the situation because it obscured Paul's position on the law when it really needed to be clarified. The Jewish Christians were not fooled, so when Paul went into the temple, they stirred up and the entire city (Acts 21:27,30) leading to Paul's arrest. He spent several years in prison and was eventually transferred to Rome. This arrest constrained his ministry, and he never got to Spain. Acts 28:17-30 puts a good spin on it, but Paul never got back into real apostolic ministry.

Following James advice brought Paul's ministry to a screaming halt. I am not sure if this was what the Holy Spirit wanted. I don't know why Paul agreed to participate in Jewish customs that he did not approve. It just increased the confusion about the relationship between the Torah and the gospel. I suspect that he was already feeling intense pressure from the Judaizers who were disrupting his work in Europe. Due to this pressure, he seems to have been intimidated by James into doing something foolish. I do not judge him for that, because we have all made that mistake, but in Paul's case, the cost was great. It seems to be a great waste of an amazing talent.

Why

I am not sure why Paul wanted to go to Jerusalem at all. He obviously wanted to make peace with the mother church, but the prophets he met on the way had warned that he would fail. I suspect God needed to take Paul out of the way so that the young church did not Paul into a pope.

Anyway, as he travelled back to Jerusalem, he was warned in various places that this visit would end with his arrest and imprisonment.

And now I am on my way to Jerusalem, compelled by the Spirit, not knowing what I will encounter there, except that in every town the Holy Spirit warns me that chains and afflictions are waiting for me (Acts 20:22-23).
When he got to Caesarea, the warning was even more specific. A prophet named Agabus tied Paul's belt around his feet, and said,
The Holy Spirit says, "In this way the Jewish leaders in Jerusalem will bind the owner of this belt and will hand him over to the Gentiles" (Acts 21:11).
His companions wanted Paul to stay away from Jerusalem, but Paul was not dissuaded.
I am ready not only to be bound, but also to die in Jerusalem for the name of the Lord Jesus (Acts 21:13).
Clearly, the Holy Spirit warned Paul that he would be arrested, but also encouraged him to go on regardless. Why did the Holy Spirit lead Paul into a situation where he would be imprisoned for several years? This would place a dampener on his ministry and limit his ability to preach the gospel and strengthen churches. It would prevent him from taking the gospel to Spain as he had planned.

My guess is that the reason the Holy Spirit allowed Paul to be imprisoned was that his ministry had been so effective that he was in danger of becoming an archbishop or mini-pope. He had planted numerous churches. The believers in those churches look up to him as their leader. He was skilled at resolving theological disputes. If he had continued in his ministry as it was, he was in danger of being raised up to a place that God did not want him to be, and where he did not want to be. (Of course, going on to Spain would have had the same effect).

An apostle must keep moving on, or they will be quickly be turned into a bishop, CEO, or Big Man, depending on the culture. Paul understood this well. He was an amazingly talented person, but the kept pressing on to new places, so the churches he established could not make him into the "archbishop" of a group of churches.

Once Paul was imprisoned, and eventually killed, the young church had to stand on their own feet and rely on the Holy Spirit. The numerous people who had travelled and worked with Paul had to step up and preach the gospel and share the truth. Others would have to take the gospel to Spain. That is how the Holy Spirit prefers to work. He prefers working through a multitude of ordinary people to relying on a few superstars.