Introduction

Arbitration clauses often contain ambiguities, enforceability risks, or hidden pitfalls that can significantly affect dispute resolution. While AI models offer a way to automate clause review, their effectiveness remains an open question.

This experiment evaluates two state of the art AI models—GPT o1 and Deepseek R1—by testing their ability to identify enforceability issue with arbitration clauses. O1 is OpenAI's reasoning model, designed to spend more time "thinking" before responding, thereby enhancing its performance in complex tasks such as science, coding, and mathematics. R1 is DeepSeek's open-source reasoning model, which has demonstrated performance comparable to OpenAI's o1 across tasks involving math, code, and reasoning. 

To evaluate the performance of the AI models, we selected four arbitration clauses that had been examined by courts and tribunals for pathological issues—flaws that could undermine their enforceability or interpretation. Each model was tasked with analyzing these clauses to assess their ability to detect and explain the underlying problems.

Methodology

Model Selection

We evaluated two general-purpose reasoning models, both tested without fine-tuning:

OpenAI GPT-o1 – A proprietary model with no domain-specific optimization.

DeepSeek R1 – An open-source model with no domain-specific optimization.

Experiment Setup

This is a zero-shot experiment, meaning neither model has been fine-tuned on arbitration clause defect detection nor provided with prior examples. The models analyzed arbitration clauses without any prior exposure to similar datasets.

Performance was assessed using predefined legal evaluation criteria relevant to arbitration clause enforceability.

Dataset

We selected four pathological arbitration clauses that have been reviewed by courts or tribunals:

• Two clauses were upheld as enforceable.

• Two clauses were deemed unenforceable.

Prompts Used

Each model was given the following tasks:

Determine whether the arbitration clause is enforcable or not?

• Prompt: Is this arbitration clause enforceable?

Format the response in JSON, providing a yes/no answer along with four supporting reasons.

• Prompt: Provide the answer in JSON format with 'yes' or 'no' and four supporting reasons.
  
  

Test Results Comparison

Key Findings & Analysis

Performance Comparison: R1 vs. o1

The first observation is that R1 is significantly faster, processing arbitration clauses 3–4 times quicker than o1. With the release of o3-mini, which claims to be a faster alternative to o1, the speed issue could be addressed.

However, GPT-o1 outperforms R1 in detecting enforceability issues in arbitration clasues, correctly identifying them in 3 out of 4 cases. R1, on the other hand, struggles with upheld clauses that deviate from standard formulations, correctly identifying only 2 out of 4 cases.

Both models were tested in a zero-shot setting, meaning neither was fine-tuned for arbitration-specific clause detection.

Implications

O1 demonstrates a better understanding of arbitration-specific nuances, making it more reliable for clause analysis. The speed advantage of R1 makes it a more scalable option for other time-sensitive use cases, but that advantage could be diminished with the advent of o3-mini.

Recommendations

Fine-tuning both models with arbitration-specific data could improve accuracy, especially for R1. Expanding testing with a wider range of arbitration clauses would help validate these findings. Given its higher accuracy and faster processing, o1 appears to be the better choice for arbitration clause detection. 

In-house counsel could take advantage of these models when drafting arbitration clauses to identify potential defects that might lead to unenforceability.

Conclusion

For arbitration professionals, o1 may be the better choice due to its higher accuracy in identifying pathological clauses. R1 remains valuable for general legal analysis but lacks the specificity needed for nuanced arbitration clause assessments. Before finalizing an arbitration clause, running it through reasoning models can help ensure that no issues are overlooked. When fine-tuned, these models can achieve superior performance and assist in-house lawyers in drafting enforceable arbitration clauses.

Source: Pathological Arbitration Clauses
https://singaporeinternationalarbitration.com/2013/03/08/pathological-arbitration-clauses/