Skip to content
LATEST
Technology

Salesforce engineers cut 600-failure triage to 90 minutes

The internal workflow system combines deterministic automation with AI interpretation to route failures from a SOAP API retirement across engineering teams.

Salesforce engineers cut 600-failure triage to 90 minutes

Salesforce said in a Nov. 17 engineering blog post that an internal AI tool helped one engineer triage more than 600 test failures in about 60 to 90 minutes.

The company said comparable manual work for one 600-failure work item had been estimated at two or three days for several engineers, making the tool an internal test of AI-assisted workflow automation.

The failures were linked to Salesforce’s retirement of its Simple Object Access Protocol login mechanism, known as SOAP API login, which had supported an older authentication model. The company said the change intentionally broke tests across teams and services that still depended on that behavior.

“One wrong classification could bury a real defect inside the blast radius,” Shireen Nagdive, engineering manager at Salesforce, said in the company’s engineering blog post.

Salesforce said the previous process required engineers to inspect stack traces, or error records showing where software failed, determine ownership and reassign failures through an interface that could handle only 50 failures at a time. “The existing workflow turned every failure into a sequence of manual decisions and individual clicks,” the blog post said.

The team built AI-Assisted Test Failure Buddy using Model Context Protocol, a method for letting AI systems interact with external tools and data. Salesforce said the tool runs as a Python and FastMCP process exposing eight tools to Claude Code or Cursor, and connects to APIs behind GUS, its internal system for work items and test failures.

The design separates judgment from execution, according to Salesforce. Deterministic code handles API calls, identifiers, validation, batching, pattern matching and reassignment mechanics, while AI is used to interpret unfamiliar stack traces, review unmatched clusters and help resolve ambiguous ownership.

That division matters because bulk reassignment can scale mistakes as easily as it scales work. Salesforce said the tool assigns related failures to a specified existing work item, uses explicit failure IDs and lets engineers apply a shared message explaining the cause and mitigation across a group.

The company said the SOAP login preset can recognize known patterns associated with the enforcement change, while AI-assisted analysis is reserved for cases that do not match expected categories. Engineers remain responsible for reviewing the tool’s conclusions and deciding whether proposed actions are correct.

Salesforce also said the team used Claude Code to help build the tool, starting from a markdown file describing the problem, design and required behavior, then generating and refining the implementation with human review and testing.

Adoption remains limited in the information released. Salesforce said the tool has been used by about three or four teams and does not yet include adoption telemetry, though it said Slack responses from other engineers indicated the workflow was helping them manage test failures.

Source: corporate blog, Salesforce Engineering, Nov. 17

More stories

Latest news