Introduce the `generate-test` command to auto-generate Jest test files using AI #1088

New issue

Open

opened 2025-10-14 16:49:24 -06:00 by navan · 0 comments

navan commented

2025-10-14 16:49:24 -06:00

Owner

Originally created by @eyaltoledano on 3/29/2025

"During our development workflow, when we start working on a task or subtask, we need a quick and reliable way to reference expected behaviors and validate the completion of tasks. Automatically generating Jest tests using AI can significantly improve context understanding and provide a robust method to ensure task completion, app stability, and prevent regressions."

Motivation

Manual test creation and validation can lead to inconsistencies and overlooked errors, reducing productivity and app stability. By integrating AI-driven automatic Jest test generation into our development workflow, we ensure comprehensive context for developers, reliable validation for task completion, and overall application stability. This approach significantly enhances AI-powered development tools like Cursor and Lovable by allowing AI to validate its work, maintaining quality and preventing regressions.

Proposed Solution

Create a new generate-test command within the Task Master CLI leveraging Claude API, which automatically generates Jest test files based on task descriptions and subtasks, providing clear validation of expected behavior.

Key functionalities:

Command invocation options:
- task-master generate-test --id=<task-id> (specific task)
- task-master generate-test --all (all tasks)
- task-master generate-test --id=<task-id> --with-subtasks (specific task and its subtasks)
- task-master generate-test --all --with-subtasks (all tasks and their subtasks)
- --prompt can be passed in any case to add additional context/ requests to influence the test generation logic or focus on something specific
Interactive prompt when no arguments provided
AI-driven analysis and prompt construction via Claude API
Automatic Jest test file generation (task_XXX.test.ts and task_XXX_YYY.test.ts)
Detailed test cases directly derived from task descriptions
Handling both parent tasks and subtasks
Robust error handling and user feedback
Introspection of current Jest test output (npm run test) to contextually inform AI about existing test coverage and dependencies, ensuring logically coherent and interconnected tests

High-Level Deliverables

Set up new CLI command (generate-test).
Implement interactive prompt when arguments are missing.
Analyze task data and construct effective prompts for Claude API.
Introspect current Jest tests (npm run test) to inform AI of existing coverage and dependencies.
Generate Jest test files from AI responses, considering existing tests, task files and --prompt value if it is present.
Validate and format test files to Jest standards.
Ensure seamless integration into existing workflow.
Integrate and generalize existing tests.mdc rule.
Update dev_workflow and README to include testing guidelines.
Provide robust feedback and error handling in CLI.
Update CLI documentation and help systems.
Update our own test suite to test the generate-test command and flags.

Key Elements

CLI Command:
- Supports options --id, --all,--with-subtasks and --prompt
- Interactive prompts for missing arguments
- Comprehensive validation and informative feedback
Claude API Integration:
- Accurate task-context prompts
- Test code generation directly from AI analysis
Jest Test File Generation:
- Jest-compatible structure
- Supports clear file naming for tasks (task_XXX.test.ts) and subtasks (task_XXX_YYY.test.ts)
Test Introspection:
- Analyze and utilize existing Jest tests for context
- Ensure logical coherence and interdependencies between tests
Task Introspection
- Analyze the task's file (which includes the whole task and subtasks) to determine what the test suite needs to include
- Each task gets its own test suite, but the suite should contain as many tasks as needed to meet the complexity of the task, which can be accessed in the complexity-report.txt
Testing Guidelines Documentation:
- Adapt existing tests.mdc rule to general use
- Adjust the init.js workflow to include the tests.mdc rule as well.
  - Adjust the .windsurfrules to include this as well
- Include testing commands, methods and guidelines in dev_workflow and README documentation
Stability and AI-Validation:
- Enables AI self-validation and regression prevention
- Improves stability and quality assurance in AI-driven environments

Example Workflow

$ task-master generate-test --id=12
→ Generating Jest test for Task ID 12...
✓ Jest test successfully generated: task_012.test.ts

$ task-master generate-test --id=12 --prompt="ensure tests cover what happens if the stripe transaction fails"
→ Generating Jest test for Task ID 12...
✓ Jest test successfully generated: task_012.test.ts

$ task-master generate-test --all --with-subtasks
→ Generating Jest tests for all tasks and subtasks...
✓ Jest tests successfully generated.

$ task-master generate-test --all --with-subtasks
→ Generating Jest tests for all tasks and subtasks...
✓ Jest tests successfully generated.

Implementation Considerations

Ensure robust and clear integration with Claude API
Compatibility and consistency with existing CLI commands
Maintainability and clarity with TypeScript
Strong error handling for network, API, and file system issues
Efficient handling and parsing of Jest test outputs for context introspection

Out of Scope (Future Considerations)

Support for additional testing frameworks
Web-based or IDE-specific UI integrations

*Originally created by @eyaltoledano on 3/29/2025* > "During our development workflow, when we start working on a task or subtask, we need a quick and reliable way to reference expected behaviors and validate the completion of tasks. Automatically generating Jest tests using AI can significantly improve context understanding and provide a robust method to ensure task completion, app stability, and prevent regressions." ### Motivation Manual test creation and validation can lead to inconsistencies and overlooked errors, reducing productivity and app stability. By integrating AI-driven automatic Jest test generation into our development workflow, we ensure comprehensive context for developers, reliable validation for task completion, and overall application stability. This approach significantly enhances AI-powered development tools like Cursor and Lovable by allowing AI to validate its work, maintaining quality and preventing regressions. ### Proposed Solution Create a new `generate-test` command within the Task Master CLI leveraging Claude API, which automatically generates Jest test files based on task descriptions and subtasks, providing clear validation of expected behavior. **Key functionalities:** - Command invocation options: - `task-master generate-test --id=<task-id>` (specific task) - `task-master generate-test --all` (all tasks) - `task-master generate-test --id=<task-id> --with-subtasks` (specific task and its subtasks) - `task-master generate-test --all --with-subtasks` (all tasks and their subtasks) - `--prompt` can be passed in any case to add additional context/ requests to influence the test generation logic or focus on something specific - Interactive prompt when no arguments provided - AI-driven analysis and prompt construction via Claude API - Automatic Jest test file generation (`task_XXX.test.ts` and `task_XXX_YYY.test.ts`) - Detailed test cases directly derived from task descriptions - Handling both parent tasks and subtasks - Robust error handling and user feedback - Introspection of current Jest test output (`npm run test`) to contextually inform AI about existing test coverage and dependencies, ensuring logically coherent and interconnected tests ### High-Level Deliverables 1. Set up new CLI command (`generate-test`). 2. Implement interactive prompt when arguments are missing. 3. Analyze task data and construct effective prompts for Claude API. 4. Introspect current Jest tests (`npm run test`) to inform AI of existing coverage and dependencies. 5. Generate Jest test files from AI responses, considering existing tests, task files and `--prompt` value if it is present. 6. Validate and format test files to Jest standards. 7. Ensure seamless integration into existing workflow. 8. Integrate and generalize existing `tests.mdc` rule. 9. Update `dev_workflow` and README to include testing guidelines. 10. Provide robust feedback and error handling in CLI. 11. Update CLI documentation and help systems. 12. Update our own test suite to test the `generate-test` command and flags. ### Key Elements - **CLI Command**: - Supports options `--id`, `--all`,`--with-subtasks` and `--prompt` - Interactive prompts for missing arguments - Comprehensive validation and informative feedback - **Claude API Integration**: - Accurate task-context prompts - Test code generation directly from AI analysis - **Jest Test File Generation**: - Jest-compatible structure - Supports clear file naming for tasks (`task_XXX.test.ts`) and subtasks (`task_XXX_YYY.test.ts`) - **Test Introspection**: - Analyze and utilize existing Jest tests for context - Ensure logical coherence and interdependencies between tests - **Task Introspection** - Analyze the task's file (which includes the whole task and subtasks) to determine what the test suite needs to include - Each task gets its own test suite, but the suite should contain as many tasks as needed to meet the complexity of the task, which can be accessed in the `complexity-report.txt` - **Testing Guidelines Documentation**: - Adapt existing `tests.mdc` rule to general use - Adjust the `init.js` workflow to include the `tests.mdc` rule as well. - Adjust the `.windsurfrules` to include this as well - Include testing commands, methods and guidelines in `dev_workflow` and README documentation - **Stability and AI-Validation**: - Enables AI self-validation and regression prevention - Improves stability and quality assurance in AI-driven environments ### Example Workflow ```shell $ task-master generate-test --id=12 → Generating Jest test for Task ID 12... ✓ Jest test successfully generated: task_012.test.ts $ task-master generate-test --id=12 --prompt="ensure tests cover what happens if the stripe transaction fails" → Generating Jest test for Task ID 12... ✓ Jest test successfully generated: task_012.test.ts $ task-master generate-test --all --with-subtasks → Generating Jest tests for all tasks and subtasks... ✓ Jest tests successfully generated. $ task-master generate-test --all --with-subtasks → Generating Jest tests for all tasks and subtasks... ✓ Jest tests successfully generated. ``` ### Implementation Considerations - Ensure robust and clear integration with Claude API - Compatibility and consistency with existing CLI commands - Maintainability and clarity with TypeScript - Strong error handling for network, API, and file system issues - Efficient handling and parsing of Jest test outputs for context introspection ### Out of Scope (Future Considerations) - Support for additional testing frameworks - Web-based or IDE-specific UI integrations