feat: Unified API key management and auditing for CLI and MCP #198

New issue

Open

opened 2025-10-14 15:45:33 -06:00 by navan · 0 comments

navan commented

2025-10-14 15:45:33 -06:00

Owner

Originally created by @thisisbbc on 8/7/2025

The current implementation of API keys, specifically the output of the API Keys Status when running 'tm models', can be very confusing and sometime adds unnecessary friction that could be avoided by a more comprehensive key manager or auditing tool.

Motivation

Some functions will fail/succeed depending if they are called from the AI chat box (Cursor, Cline) or from CLI.

Environment variables can be configured locally per project, globally or via oAuth (Gemini CLI, Claude Code).
MCP configuration can also be set locally per project or globally (e.g. via the IDE settings).

It is sometimes difficult to know where the issue originated from when configuration can change from one project to another and is scattered in various folders and locations.

Proposed Solution

Structurally speaking, I'm not sure what would be the best approach to fix this.
However, I think some sort of 'audit' tool could help to identify when and where issues arise.

What I mean by an audit tool is a function that would run tests on the Main, Research and Fallback models configured, both via Environment variables, MCP and oAuth (for claude-code and gemini-cli), to ensure all vectors are properly tested.

Something like 'tm models --audit'

High-Level Workflow

User run 'tm init' and configure desired models
User run 'tm models' to confirm Main, Research and Fallback models are configured
User run 'tm models --audit' to test the configured models are working properly

Key Elements

Running 'tm models --audit' allows users to test all models without modifying any project files
The output of the function would provide comprehensive information about the key, key type (CLI Key, MCP Key), key location and success/failure test state

Example Workflow

Provide a clear, concrete example demonstrating the feature:

$ task-master models --audit
→ Running audit on configured keys...
→ Main, Research and Fallback models configured
→ Main: gemini-2.5-pro-preview-05-06 (Google)
→ → → CLI Key: AIzaSyAofeskuhseEh5d2gesglihKe5e5JtvnWbURqy45ete
→ → → CLI Key Location: ./.env
→ → → CLI Key Status: Working
→ → → MCP Key: AIzaSyAofeskuhseEh5d2gesglihKe5e5JtvnWbURqy45ete
→ → → MCP Key Location: ./.taskmaster/mcp.json
→ → → MCP Key Status: Working
→ → → Summary: 2/2 authentification method working. You can use both CLI and MCP.
→ Research: opus (claude-code)
→ → → CLI Key: your_anthropic_api_key_here
→ → → CLI Key Location: ./.env
→ → → CLI Key Status: Failure! Please verify configuration
→ → → MCP Key: sk-ant-api03-ajkhaksjhdBkgkdsgjndsgknDlkngsl_ds45g5g6s1egDkj
→ → → MCP Key Location: C:/Users/myUsername/.claude/credentials.json
→ → → MCP Key Status: Working
→ → → oAuth Status: Authenticated
→ → → Summary: 2/3 authentification method working. You can use both CLI and MCP.
→ Fallback: claude-3-5-sonnet-20241022 (Anthropic)
→ → → CLI Key: your_anthropic_api_key_here
→ → → CLI Key Location: ./.env
→ → → CLI Key Status: Failure! Please verify configuration
→ → → MCP Key: sk-ant-api03-ajkhaksjhdBkgkdsgjndsgknDlkngsl_ds45g5g6s1egDkj
→ → → MCP Key Location: C:/Users/myUsername/.cursor/mcp.json
→ → → MCP Key Status: Working
→ → → Summary: 1/2 authentification method working. You can only use MCP.

*Note: All API keys above are obviously not proper keys and typed randomly
*Note2: The Working/Failure status should be based on an actual test API call, not just if the key is found/not found

Out of Scope (Future Considerations)

Maybe an idea for a future VS extension (or bundled into the existing Kanban Board extension) that would provide a more comprehensive GUI for key management and auditing.

*Originally created by @thisisbbc on 8/7/2025* The current implementation of API keys, specifically the output of the API Keys Status when running 'tm models', can be very confusing and sometime adds unnecessary friction that could be avoided by a more comprehensive key manager or auditing tool. ### Motivation Some functions will fail/succeed depending if they are called from the AI chat box (Cursor, Cline) or from CLI. Environment variables can be configured locally per project, globally or via oAuth (Gemini CLI, Claude Code). MCP configuration can also be set locally per project or globally (e.g. via the IDE settings). It is sometimes difficult to know where the issue originated from when configuration can change from one project to another and is scattered in various folders and locations. ### Proposed Solution Structurally speaking, I'm not sure what would be the best approach to fix this. However, I think some sort of 'audit' tool could help to identify when and where issues arise. What I mean by an audit tool is a function that would run tests on the Main, Research and Fallback models configured, both via Environment variables, MCP and oAuth (for claude-code and gemini-cli), to ensure all vectors are properly tested. Something like 'tm models --audit' ### High-Level Workflow 1. User run 'tm init' and configure desired models 2. User run 'tm models' to confirm Main, Research and Fallback models are configured 3. User run 'tm models --audit' to test the configured models are working properly ### Key Elements - Running 'tm models --audit' allows users to test all models without modifying any project files - The output of the function would provide comprehensive information about the key, key type (CLI Key, MCP Key), key location and success/failure test state ### Example Workflow Provide a clear, concrete example demonstrating the feature: ```shell $ task-master models --audit → Running audit on configured keys... → Main, Research and Fallback models configured → Main: gemini-2.5-pro-preview-05-06 (Google) → → → CLI Key: AIzaSyAofeskuhseEh5d2gesglihKe5e5JtvnWbURqy45ete → → → CLI Key Location: ./.env → → → CLI Key Status: Working → → → MCP Key: AIzaSyAofeskuhseEh5d2gesglihKe5e5JtvnWbURqy45ete → → → MCP Key Location: ./.taskmaster/mcp.json → → → MCP Key Status: Working → → → Summary: 2/2 authentification method working. You can use both CLI and MCP. → Research: opus (claude-code) → → → CLI Key: your_anthropic_api_key_here → → → CLI Key Location: ./.env → → → CLI Key Status: Failure! Please verify configuration → → → MCP Key: sk-ant-api03-ajkhaksjhdBkgkdsgjndsgknDlkngsl_ds45g5g6s1egDkj → → → MCP Key Location: C:/Users/myUsername/.claude/credentials.json → → → MCP Key Status: Working → → → oAuth Status: Authenticated → → → Summary: 2/3 authentification method working. You can use both CLI and MCP. → Fallback: claude-3-5-sonnet-20241022 (Anthropic) → → → CLI Key: your_anthropic_api_key_here → → → CLI Key Location: ./.env → → → CLI Key Status: Failure! Please verify configuration → → → MCP Key: sk-ant-api03-ajkhaksjhdBkgkdsgjndsgknDlkngsl_ds45g5g6s1egDkj → → → MCP Key Location: C:/Users/myUsername/.cursor/mcp.json → → → MCP Key Status: Working → → → Summary: 1/2 authentification method working. You can only use MCP. ``` *Note: All API keys above are obviously **not** proper keys and typed randomly *Note2: The Working/Failure status should be based on an actual test API call, not just if the key is found/not found ### Out of Scope (Future Considerations) Maybe an idea for a future VS extension (or bundled into the existing Kanban Board extension) that would provide a more comprehensive GUI for key management and auditing.