Troubleshooting AI issues

How to fix common problems when LOIS for Word Composer isn’t giving the contract review results you need, with diagnostic prompts and refinement strategies.

When AI isn’t working as expected, the problem usually falls into a few predictable categories. This guide helps you diagnose and fix common issues quickly.

Inconsistent results

Problem: Different output each time

You run the same prompt on the same document but get different results.

Common causes:

• Prompt is too vague or open-ended
• Missing essential context
• Relying on AI to infer rather than instruct

Solutions:

Add specific constraints: "Focus only on payment terms"
Include clear context: "We are the vendor with low leverage"
Define output format: "List exactly 3 issues in order of importance"

Test: Run the improved prompt 3 times. Results should be nearly identical.

Problem: Quality varies wildly

Sometimes the AI nails it, sometimes it’s completely off.

Common causes:

• Prompt works for some document types but not others
• Context requirements vary by situation
• Model updates affecting performance

Solutions:

• Create document-specific prompts rather than universal ones
• Add conditional logic: “If customer is government entity, then…”
• Test prompts regularly and update as needed

Wrong focus

Problem: AI focuses on wrong issues

The AI obsesses over formatting while missing major liability issues.

Common causes:

• No priority guidance provided
• Equal weight given to all issues
• Missing risk framework

Solutions:

"Ignore formatting and stylistic issues"
"Focus only on provisions creating financial risk over $10,000"
"Prioritize: 1) Liability 2) Payment 3) IP rights - ignore everything else"

Problem: Too much or too little detail

Getting novels when you need summaries, or single sentences when you need analysis.

Common causes:

• No length specification
• Audience unclear
• Purpose undefined

Solutions:

"One sentence per issue"
"Detailed analysis with specific clause references"
"Executive summary: 3 bullet points maximum"

Context problems

Problem: AI doesn’t understand the situation

The AI gives generic advice that doesn’t fit your specific circumstances.

Common causes:

• Insufficient context provided
• Wrong context emphasized
• Contradictory information given

Solutions: Build context systematically:

1. Who you are: "Small startup vendor"
2. Document type: "Enterprise MSA"
3. Leverage: "Very low - competitive RFP"
4. Constraints: "Cannot accept uncapped liability per insurance"

Problem: AI ignores critical context

You mentioned you’re in healthcare but the AI ignores HIPAA requirements.

Common causes:

• Context buried in long prompt
• Competing contexts confusing priority
• Key context mentioned too late

Solutions:

• Lead with critical context
• Repeat important constraints
• Use headers to organize context clearly

Output format issues

Problem: Wrong format for needs

Getting prose when you need a table, or bullets when you need an email.

Common causes:

• Format not specified
• Conflicting format instructions
• AI defaulting to its preferences

Solutions: Be explicit about format:

"Create a table with these exact columns: Issue | Risk | Action"
"Draft as email suitable for sending to opposing counsel"
"Bullet points only - no prose paragraphs"

Problem: Information scattered

Important points buried throughout long responses.

Common causes:

• No structure requested
• Multiple asks in one prompt
• AI trying to be comprehensive

Solutions:

"Organize by: 1) Critical issues 2) Medium priority 3) Minor points"
"Start with your recommendation, then explain"
"Group all payment-related issues together"

Accuracy issues

Problem: AI misses important provisions

The AI doesn’t catch critical clauses.

Common causes:

• Rule/prompt too narrow
• Keywords not matching document language
• Important sections in unexpected places

Solutions:

• Broaden search terms: “audit, inspection, review, examination”
• Check entire document: “Review all sections including exhibits”
• Be explicit: “Check for liability caps anywhere in document”

Problem: False positives

AI flags issues that aren’t really problems.

Common causes:

• Prompt too broad
• Missing negative prompting
• No threshold for materiality

Solutions:

"Only flag if liability exceeds $100,000"
"Ignore standard industry terms"
"Do not flag: payment terms, notice provisions, standard definitions"

Performance issues

Problem: AI takes forever

Response times are unusually long.

Common causes:

• Overly complex prompts
• Multiple large documents
• Requesting too much in one prompt

Solutions:

• Break into smaller requests
• Process documents separately
• Simplify prompt structure

Problem: AI cuts off mid-response

Responses end abruptly without completing the task.

Common causes:

• Output length limits reached
• Token limits exceeded
• Complex chains causing timeouts

Solutions:

• Request more concise output
• Break into multiple prompts
• Ask for summary first, details separately

Model-specific issues

Problem: Prompt works in one model but not another

Different AI models behave differently.

Common causes:

• Models trained differently
• Format preferences vary
• Instruction interpretation differs

Solutions:

• Adjust formatting for the specific model you’re using
• Test prompts on your target platform
• Use model-appropriate structure (some models prefer markdown headers, others respond better to XML tags)

Problem: Quality degraded after update

Previously good prompts now failing.

Common causes:

• Model improvements changing behavior
• Prompt exploiting previous quirks
• Overfitting to old model version

Solutions:

• Simplify and clarify prompts
• Remove workarounds for old issues
• Test and update prompt library

Quick diagnostic checklist

When something’s not working, check:

1. Context: Is it clear who you are and what you need?
2. Specificity: Are instructions precise or vague?
3. Format: Did you specify how you want output?
4. Scope: Is the AI looking at the right things?
5. Constraints: Are there clear boundaries?
6. Priority: Does the AI know what matters most?

Iterative refinement

When a prompt isn’t working, resist the urge to rewrite it from scratch. Small, targeted adjustments are faster and more effective.

Step 1: Identify the gap

Run your prompt and compare the output to what you wanted. Ask: is the problem with context, scope, format, or specificity?

Step 2: Make one change at a time

Adjust a single element per iteration so you can tell what fixed it:

Attempt 1: "Review this agreement for risks."
→ Too broad. Add scope.

Attempt 2: "Review this agreement for risks related to liability and indemnification."
→ Right focus, but too much detail. Add constraints.

Attempt 3: "Review this agreement for risks related to liability and indemnification. List the top 3 issues only."
→ Getting there. Add context.

Attempt 4: "We are the vendor with low leverage. Review this agreement for risks related to liability and indemnification. List the top 3 issues only."
→ Focused, contextual, constrained.

Step 3: Save what works

Once you land on a prompt that delivers, save it to your prompt library with notes on what you changed and why.

Emergency fixes

Start completely fresh with the simplest possible version:

"I'm the vendor. Find the three biggest risks in this agreement."

Then gradually add complexity back.

Use pre-built templates or playbooks rather than custom prompts. Even if not perfect, they’re tested and reliable.

If AI suggests something that seems legally problematic, always verify against source material and apply professional judgment.

Prevention strategies

• Test early and often Don’t wait until the big deal to discover your prompt doesn’t work.
• Document what works When you find a good prompt, save it immediately with notes on why it worked.
• Build gradually Start simple and add complexity only as needed.
• Stay updated AI models evolve. What worked last month might need adjustment today.

Remember

Most AI problems stem from communication issues, not AI limitations. The AI can’t read your mind – it only knows what you tell it. When troubleshooting, start by examining your prompt before blaming the model.

Clear instructions beat clever prompts every time.