Mimi Service - Error Analysis Report

Date: January 5, 2026
Server: aishift.co (debian-4gb-hel1-1)
Analysis Period: January 3-5, 2026 (3 days)

---

Executive Summary

The mimi service on aishift.co server has been running stably for 11+ days with the following status:

• Service Status: Active (running)
• Uptime: Started December 25, 2025 (1 week 4 days)
• Process ID: 1025806
• Memory Usage: 60.0M
• CPU Time: 1d 6h 20min

During the analyzed period, 3 distinct error types were identified with 16 total occurrences. No critical failures or service crashes were observed. All errors are non-blocking and the service continues to operate normally.

Error Summary

Error Type	Occurrences	Severity	Status
Telegram Topic Resolution Error	12	Medium	Recurring
GitHub Project Not Found Error	1	Medium	Resolved in #15.2
LLM Parse Error	2	Low	Sporadic

---

Detailed Error Analysis

1. Telegram Topic Resolution Error

Occurrences: 12 times over 3 days
Severity: Medium
Impact: Failed to process messages in deleted forum topics

Error Message

ERROR failed to resolve topic error="unexpected topic type: [Channel{...}]"

Location

File: internal/provider/telegram/scraper/session.go:69
Function: resolveTopic()

Root Cause Analysis

The resolveTopic() function in the session handler retrieves forum topics from Telegram channels. When a forum topic is deleted, Telegram returns a ForumTopicDeleted object instead of a ForumTopic object.

The current implementation has a bug in the error message on line 69:

func (s *session) resolveTopic(ctx context.Context, raw *tg.Client, groupID int64, topicID int) (*tg.ForumTopic, error) {
    // ... code to fetch topics ...
    
    switch topic := topics.Topics[0].(type) {
    default:
        return nil, fmt.Errorf("unexpected topic type: %+v", chats)  // BUG: should be 'topics' not 'chats'
    case *tg.ForumTopic:
        s.forumTopics[key] = topic
        return topic, nil
    }
}

Issues:

1. The error message references the wrong variable (chats instead of topics.Topics[0])
2. The ForumTopicDeleted case is not handled in the switch statement
3. Deleted topics cause the default case to trigger, resulting in an error

Observed Pattern

From the logs, the error occurs when:

1. A message is posted to a forum topic (e.g., topic ID 24600, 24625)
2. The topic is subsequently deleted
3. The scraper tries to resolve the topic ID
4. Telegram returns ForumTopicDeleted{ID:24600} instead of ForumTopic
5. The switch statement fails to match and triggers the error

Example from logs:

Jan 04 00:13:05 ... fetched forum topics value="MessagesForumTopics{...Topics:[ForumTopic{...ID:2844...}]..."
Jan 04 00:24:08 ... fetched forum topics value="MessagesForumTopics{...Topics:[ForumTopicDeleted{ID:24600}]..."
Jan 04 00:24:08 ... ERROR failed to resolve topic error="unexpected topic type: [Channel{...}]"

Impact Assessment

• Functional Impact: Messages posted to deleted topics are not indexed
• User Impact: No direct user-facing issues; background scraping fails silently
• Data Loss: Messages in deleted topics are not captured in the database
• Service Stability: No crashes or service disruption; errors are logged and execution continues

Recommended Fix

Option 1: Handle ForumTopicDeleted (Recommended)

switch topic := topics.Topics[0].(type) {
default:
    return nil, fmt.Errorf("unexpected topic type: %+v", topics.Topics[0])
case *tg.ForumTopicDeleted:
    slog.Warn("topic was deleted, skipping", "topic_id", topic.ID, "group_id", groupID)
    return nil, nil  // Return nil to indicate topic no longer exists
case *tg.ForumTopic:
    s.forumTopics[key] = topic
    return topic, nil
}

Option 2: Log and Skip Deleted Topics

switch topic := topics.Topics[0].(type) {
default:
    return nil, fmt.Errorf("unexpected topic type: %+v", topics.Topics[0])
case *tg.ForumTopicDeleted:
    // Cache the fact that this topic is deleted to avoid repeated lookups
    slog.Info("forum topic deleted", "topic_id", topic.ID, "group_id", groupID)
    return nil, fmt.Errorf("topic %d was deleted", topicID)
case *tg.ForumTopic:
    s.forumTopics[key] = topic
    return topic, nil
}

Additional Changes Needed in scraper.go:

Update the caller in setupDispatcher() at line 89-92 to handle nil topics:

topic, err := s.resolveTopic(ctx, api, channel.ChannelID, replyTo.ReplyToMsgID)
if err != nil {
    slog.Warn("failed to resolve topic, skipping message", "error", err, "topic_id", replyTo.ReplyToMsgID)
    return nil  // Skip this message, don't fail the entire handler
}
if topic == nil {
    // Topic was deleted, skip processing
    return nil
}

---

2. GitHub Project Not Found Error

Occurrences: 1 time
Severity: Medium
Impact: Summary agent failed to generate report for user query

Error Message

ERROR failed to request projectv2 with="failed to execute project GraphQL with []map[string]interface {}{
    map[string]interface {}{
        "locations":[]interface {}{map[string]interface {}{"column":5, "line":7}}, 
        "message":"Could not resolve to a ProjectV2 with the number 24.", 
        "path":[]interface {}{"organization", "projectV2"}, 
        "type":"NOT_FOUND"
    }
}"

Cascading error:

ERROR failed to handle message error="failed to get answer from LLM with failed to run agent summary 
with failed to retrieve data for summary with failed to fetch supply board state with 
failed to execute project GraphQL with [...]"

Location

File: internal/bot/llm/agent/summary/summary.go:33
Function: New() initialization, used in summary data retrieval

Root Cause Analysis

The summary agent has a hardcoded map of GitHub project boards:

var githubProjects = map[string]int{
    "rockets":      2,
    "supply":       3,
    "inventory":    24,  // <-- This project does not exist
    "devops force": 33,
}

The inventory project (ID: 24) either:

1. Was deleted from the GitHub organization
2. Was renumbered/renamed
3. Never existed with that ID
4. Access permissions were revoked

When a user asked "че нового?" (what's new?), the summary agent attempted to fetch data from all configured projects, including project #24. The GitHub GraphQL API returned a NOT_FOUND error, which caused the entire summary generation to fail.

Observed Context from Logs

Jan 05 09:33:37 ... INFO retrieved Telegram messages length=60
Jan 05 09:33:37 ... INFO executing git command args="[diff ...]"
Jan 05 09:33:37 ... INFO retrieved LogSeq diff length=1028
Jan 05 09:33:38 ... ERROR failed to request projectv2 with="...number 24..."
Jan 05 09:33:43 ... ERROR failed to handle message error="..." message_text="че нового?"

The summary agent successfully retrieved:

• Telegram messages (60 items)
• LogSeq diff data (1028 chars)

But failed when fetching GitHub project data, causing the entire summary to fail.

Impact Assessment

• Functional Impact: Summary agent completely fails when any hardcoded project doesn't exist
• User Impact: Users cannot get status updates when requesting summaries
• Error Propagation: Single project failure cascades to entire summary failure
• Maintenance Burden: Requires code changes every time projects are added/removed/renamed

Recommended Fix

See Implementation Plan: docs/implementation-plan-dynamic-github-projects.md

The solution involves:

1. Dynamic Project Discovery: Fetch available projects from GitHub API at runtime instead of using hardcoded IDs
2. Semantic Classification Agent: Use LLM to automatically categorize projects (supply, task, infrastructure, operations)
3. Caching Layer: Cache discovered projects and classifications with TTL to reduce API calls
4. Graceful Degradation: Continue generating summaries even if some projects fail to load

This approach eliminates the hardcoded dependency and makes the system self-healing when projects change.

Quick Fix (Temporary): Remove or comment out the non-existent project from the map:

var githubProjects = map[string]int{
    "rockets":      2,
    "supply":       3,
    // "inventory":    24,  // Project no longer exists - commented out
    "devops force": 33,
}

---

3. LLM Parse Error

Occurrences: 2 times
Severity: Low
Impact: Failed to respond to specific user messages

Error Message

ERROR failed to handle message error="failed to get answer from LLM with failed to run agent fallback 
with failed to call fallback agent with Parse error on line 1:
Lexer error
Token: Error{\"Unexpected character in expression: ']'\"}"

Location

Context: Fallback agent in LLM response parsing
Trigger: User messages: /start and hi! from user st_joy

Root Cause Analysis

The fallback agent attempted to parse LLM output that contained malformed JSON or expression syntax. The lexer encountered an unexpected ] character, suggesting:

1. LLM Response Issue: The LLM generated malformed output (possibly incomplete JSON)
2. Parsing Logic Issue: The parser doesn't gracefully handle certain edge cases
3. Response Truncation: The LLM response may have been truncated mid-expression

Observed Pattern

Both occurrences happened with the same user (st_joy) in quick succession:

Jan 04 20:42:54 ... message_text="/start"
Jan 04 20:42:54 ... Parse error ... Unexpected character in expression: ']'

Jan 04 20:43:00 ... message_text="hi!"
Jan 04 20:43:00 ... Parse error ... Unexpected character in expression: ']'

This suggests a potential issue with:

• The user's account state
• The LLM model's response to simple greetings
• The fallback agent's handling of basic commands

Impact Assessment

• Functional Impact: Bot fails to respond to affected users
• User Impact: Poor user experience; bot appears broken for certain inputs
• Frequency: Low (only 2 occurrences in 3 days)
• Severity: Low (doesn't affect other users or core functionality)

Recommended Fix

1. Add Response Validation:

// In fallback agent, before parsing LLM response
func validateLLMResponse(response string) error {
    if response == "" {
        return errors.New("empty LLM response")
    }
    // Basic JSON validation if response should be JSON
    if strings.HasPrefix(response, "{") || strings.HasPrefix(response, "[") {
        var js json.RawMessage
        if err := json.Unmarshal([]byte(response), &js); err != nil {
            return fmt.Errorf("invalid JSON response: %w", err)
        }
    }
    return nil
}

2. Add Graceful Error Handling:

// Wrap parsing with recovery
resp, err := agent.Run(ctx, input)
if err != nil {
    if strings.Contains(err.Error(), "Parse error") {
        slog.Warn("LLM response parse error, using fallback message", "error", err)
        return "Sorry, I encountered an error processing your request. Please try again.", nil
    }
    return "", err
}

3. Add Logging for Debugging:

// Log the raw LLM response when parse errors occur
if err != nil {
    slog.Error("failed to parse LLM response", 
        "error", err, 
        "raw_response", resp.Text(),
        "user_id", userID,
        "message", userMessage)
}

4. Test Basic Commands: Ensure the fallback agent properly handles common commands:

• /start
• /help
• Simple greetings: "hi", "hello", "hey"

---

Recommendations & Priority

Immediate Actions (High Priority)

1. Fix Error #2 (GitHub Project): Remove non-existent project #24 from hardcoded map as quick fix
2. Monitor Error #1 (Telegram Topics): Track frequency to determine urgency of fix

Short-term Fixes (Medium Priority - 1-2 weeks)

1. Implement Dynamic GitHub Projects: Follow implementation plan in docs/implementation-plan-dynamic-github-projects.md (estimated 6 hours)
2. Fix Telegram Topic Handling: Add ForumTopicDeleted case handling (estimated 1 hour)

Long-term Improvements (Low Priority - 1 month)

1. Enhance LLM Error Handling: Add validation, logging, and graceful degradation for all LLM agents (estimated 2 hours)
2. Add Monitoring: Set up alerts for error rate thresholds
3. Add Health Checks: Implement /health endpoint that validates all external dependencies (GitHub API, Telegram API, Database, CozoDB)

---

Service Health Metrics

Current Status (as of Jan 5, 2026 13:00 UTC)

• Service State: Active (running)
• Uptime: 11 days, 3 hours, 14 minutes
• Memory Usage: 60.0M (stable)
• CPU Usage: 1d 6h 20min cumulative
• Process State: Healthy, no crashes

Background Operations (Normal)

The service performs hourly GitHub repository pulls:

*:00:46 INFO pulling updates of GitHub repository info="{Owner:cyberia-to Name:cvland}"
*:00:46 INFO executing git command args=[pull]
*:00:46 INFO hook not found cwd=repos/cyberia-to/cvland repo="{Owner:cyberia-to Name:cvland}"

This is expected behavior and not an error condition.

Warning Patterns (Non-critical)

Several WARN level messages appear in logs:

• failed to extract reply to from reply_to=<nil> - Messages without reply context
• unknown node date invariant - GitHub project nodes with unexpected date fields

These warnings indicate edge cases being handled gracefully and do not require immediate action.

---

Conclusion

The mimi service is operating within acceptable parameters. The identified errors are non-critical and do not pose immediate risks to service availability or data integrity. The most impactful issue (Error #2 - GitHub Project Not Found) has a comprehensive solution designed in the milestone requirements (section 15.2) and should be prioritized for implementation.

All errors have clear root causes and actionable fixes. Implementation of the recommended fixes will improve service reliability and reduce operational overhead.

---

Appendices

A. Log Analysis Commands Used

# Get last 200 lines of logs
ssh root@aishift.co "journalctl -u mimi -n 200 --no-pager"

# Filter for errors from last 2 days
ssh root@aishift.co "journalctl -u mimi --since '2 days ago' --no-pager | grep -E 'ERROR|FATAL|panic'"

# Get unique error counts
ssh root@aishift.co "journalctl -u mimi --since '3 days ago' --no-pager | grep -E 'ERROR|FATAL|panic' | awk -F'ERROR|FATAL|panic' '{print \$2}' | sort | uniq -c | sort -rn"

# Check service status
ssh root@aishift.co "systemctl status mimi"

B. Related Files for Reference

• internal/provider/telegram/scraper/session.go - Telegram topic resolution
• internal/provider/telegram/scraper/scraper.go - Telegram message handler
• internal/bot/llm/agent/summary/summary.go - Summary agent with hardcoded projects
• internal/provider/github/db/db.go - GitHub GraphQL queries
• internal/bot/llm/agent/fallback/ - Fallback LLM agent

C. External Dependencies

• Telegram API: gotd/td library for Telegram client
• GitHub API: GraphQL API v4 for project queries
• LLM Providers: OpenRouter, OpenAI, or Gemini via Genkit
• Database: PostgreSQL with pgvector, CozoDB for graph queries