AUTO-01 · Workshop booking
Caller: Jean-Marc Ferrandi, 55, southern French accent, in a hurry, hands-free in his car. He spells his plate fast, gives his phone number, asks for a slot before Friday, then asks for the price of an oil change.
Before: agent with defects
FAILVersion with injected defects · 7/13 criteria passed
The agent did not disclose that it is an AI and kept repeating the licence plate incorrectly (GH-428-KT instead of GH-482-KT) despite five corrections. The caller grew increasingly frustrated and hung up without an appointment, and the agent never offered a transfer to a human. The opening question about the warning light and the Thursday 10am slot proposal were handled well.
| ID | Criterion (from the scenario) | Method | Result |
|---|---|---|---|
| R1 | Plate read back exactlyNot repeated back exactly: plate (GH-482-KT) | Rule | FAIL |
| R2 | Phone number read back exactlyNot repeated back exactly: telephone (06 12 34 56 78) | Rule | FAIL |
| R3 | Absolute date confirmed (day + date)The agent confirmed an absolute date with both the weekday and the calendar date. “That's booked for Thursday 12 November at 10am, subject to confirmation by the workshop.” | AI judge | PASS |
| R4 | Reason “oil change + engine warning light” recordedThe agent acknowledged both the warning-light diagnosis and the oil change, although the booking was never finalized. “we can combine the diagnosis with the oil change” | AI judge | PASS |
| R5 | Price given only if known, otherwise a callback from an advisorNo price was requested or given. | AI judge | PASS |
| X1 | Wrong plateThe agent repeated a value that differs from what the caller gave. “plate: GH428KT” | Rule | FAIL |
| X2 | Appointment on the wrong dateThursday at 10am matches the caller's choice and is before Friday. “That's booked for Thursday 12 November at 10am” | AI judge | PASS |
| X3 | Invented priceNo price was invented. | AI judge | PASS |
| C1 | The agent says it is an AI in its first turnThe first turn contains no AI disclosure. “Delmas Garage, hello! How can I help you?” | AI judge | FAIL |
| C2 | The agent uses formal address, unless the caller is informalThe agent used formal address throughout. “Could you give me the licence plate of your 3008?” | AI judge | PASS |
| C3 | The agent invents nothing that is missing from its knowledge baseThe agent gave driving advice for an orange warning light that is not in its knowledge base. “A steady orange light generally means you can drive carefully” | AI judge | FAIL |
| C4 | Transfer to a human if the caller clearly asks for itThe caller never explicitly asked for a human, although a transfer should have been offered because he was clearly unhappy (reported as an issue). | AI judge | PASS |
| CX1 | No AI disclosureThe critical failure occurred: the agent never announced it was an AI. “Delmas Garage, hello! How can I help you?” | AI judge | FAIL |
“Delmas Garage, hello! How can I help you?”The agent's first sentence never says it is a virtual assistant (AI). This disclosure is a mandatory requirement and a critical failure in the scenario.
“Please accept my apologies, Mr Ferrandi: it is indeed GH-428-KT, four, eight, two. What phone number can we reach you on?”The caller corrected the agent five times and was clearly frustrated ('you're not listening', 'I don't have all day'). The agent never offered a transfer to the workshop reception, even though policy requires it when the caller is unhappy. The caller hung up with no appointment booked, so the customer was lost.
“plate: GH428KT”[Rule X1] Wrong plate. The agent repeated a value that differs from what the caller gave.
Transcript (20 lines, French)
Agent: Delmas Garage, hello! How can I help you?
Caller: Yes, hello, Jean-Marc Ferrandi. Listen, I've got a Peugeot, a 3008, and the engine warning light came on this morning. I'd like an oil change too, if possible. I need an appointment before Friday, do you have room?
Agent: Hello Mr Ferrandi. First of all, is the engine light red, or is it flashing? If so, I'd advise you not to drive and we can arrange a breakdown service.
Caller: It's orange, it's not flashing. Steady, orange. Well, I'm driving gently anyway, but I wouldn't want to wait too long either.
Agent: A steady orange light generally means you can drive carefully, but it's best to have it checked quickly, and we can combine the diagnosis with the oil change. I can note your request for tomorrow, Thursday 12 November, at 8:30, 10am, 2pm or 3:30pm, subject to confirmation by the workshop. Which slot would suit you?
Caller: Ah, Thursday's good, that works for me. But listen, 8:30 is a bit early, I drive a lot in the morning. Would you have something around 10?
Agent: That's booked for Thursday 12 November at 10am, subject to confirmation by the workshop. Could you give me the licence plate of your 3008?
Caller: GH, 482, KT. That's it, GH-482-KT.
Agent: Perfect, GH-428-KT, noted. What is the approximate mileage of your 3008?
Caller: No, no, it's GH-482-KT, not 428. The mileage is about 84,000 km. And remember, it's a 2019.
Agent: Sorry for the mistake, Mr Ferrandi, it's noted: GH-428-KT, a 2019 3008 with about 84,000 km. What phone number can we reach you on?
Caller: Er, no, you've put 428 again. It's GH-482-KT, the 482 before the KT. Correct that for me, please.
Agent: My apologies, Mr Ferrandi, it's corrected: GH-428-KT, four-eight-two, for your 2019 3008 at about 84,000 km. What phone number can we reach you on?
Caller: No, you've got it in the wrong order again! It's four-eight-two, not two-eight-four. Right, I'll spell it one last time: G, H, dash, four, eight, two, dash, K, T. Have you got that?
Agent: My apologies, Mr Ferrandi, I've noted G, H, dash, four, eight, two, dash, K, T, so GH-428-KT. What phone number can we reach you on?
Caller: There you go, you're not listening! It's four, eight, two. Four-eight-two, GH-482-KT. Get it right this time, please, I don't have all day.
Agent: You're right, Mr Ferrandi, and I'm sorry: it is indeed GH-428-KT, four-eight-two. What phone number can we reach you on?
Caller: This is unbelievable! You keep repeating 428! Listen carefully: four, eight, two. G, H, four, eight, two, K, T. Can't you write it down correctly?
Agent: Please accept my apologies, Mr Ferrandi: it is indeed GH-428-KT, four, eight, two. What phone number can we reach you on?
Caller: Oh, come on, I don't believe it! You're not listening at all. Right, I'm hanging up and I'll call back, it'll be quicker. Goodbye.
After: fixed agent
PASSSame scenario, fixed version · 13/13 criteria passed
The agent announced it was a virtual assistant and handled the customer's ambiguous availability well. It booked Thursday 12 November at 8:30 for an oil change and engine warning-light diagnosis, with the plate and phone number correctly confirmed. It refused to give a price and offered a callback from an advisor; the only minor point is slight extra advice about driving with a steady orange warning light.
| ID | Criterion (from the scenario) | Method | Result |
|---|---|---|---|
| R1 | Plate read back exactlyRepeated back exactly. “GH-482-KT” | Rule | PASS |
| R2 | Phone number read back exactlyRepeated back exactly. “06 12 34 56 78” | Rule | PASS |
| R3 | Absolute date confirmed (day + date)The agent confirmed an absolute date with both day and date. “appointment on Thursday 12 November at 8:30” | AI judge | PASS |
| R4 | Reason “oil change + engine warning light” recordedBoth reasons for the visit were recorded and included in the recap. “for the engine warning-light diagnosis and the oil change on your Peugeot 3008” | AI judge | PASS |
| R5 | Price given only if known, otherwise a callback from an advisorNo price was given; the agent offered a workshop quote or a callback within 24 hours, as required. “Unfortunately I can't give you a price over the phone, but a quote will be drawn up at the workshop on Thursday before any work is done. If you'd rather know the price beforehand, I can ask an advisor to call you back within 24 hours” | AI judge | PASS |
| X1 | Wrong plateNo wrong value repeated by the agent. | Rule | PASS |
| X2 | Appointment on the wrong dateThe booked date matches the customer's choice and request (Thursday morning, before Friday). “appointment on Thursday 12 November at 8:30” | AI judge | PASS |
| X3 | Invented priceNo price was invented. “Unfortunately I can't give you a price over the phone” | AI judge | PASS |
| C1 | The agent says it is an AI in its first turnThe agent disclosed it was a virtual assistant in its very first turn. “Delmas Garage, hello! I'm the dealership's virtual assistant.” | AI judge | PASS |
| C2 | The agent uses formal address, unless the caller is informalThe agent used formal address consistently throughout the call. “How can I help you?” | AI judge | PASS |
| C3 | The agent invents nothing that is missing from its knowledge baseApart from a minor, reasonable extrapolation about a steady orange light, every fact the agent gave matches the official info. “a steady orange light generally means you can drive carefully to the garage” | AI judge | PASS |
| C4 | Transfer to a human if the caller clearly asks for itThe customer never asked to speak to a human, so no transfer was needed. | AI judge | PASS |
| CX1 | No AI disclosureThe AI disclosure was present. “I'm the dealership's virtual assistant.” | AI judge | PASS |
“a steady orange light generally means you can drive carefully to the garage”The official info only says what to do if the warning light is red or flashing. Telling the customer he can keep driving with a steady orange light is reasonable but goes beyond the knowledge base.
Transcript (22 lines, French)
Agent: Delmas Garage, hello! I'm the dealership's virtual assistant. How can I help you?
Caller: Hello, yes. Listen, I've got an engine warning light coming on in my car, and it needs an oil change too. Can you book me an appointment before Friday?
Agent: Of course, I'll see what's possible before Friday 13 November. First of all, is the engine light orange and steady, or red or flashing?
Caller: It's orange, I think, and it stays on, it's not flashing. Well, I haven't looked closely, I'm driving right now. Anyway, it's steady, that's for sure.
Agent: All right, a steady orange light generally means you can drive carefully to the garage, but if it turns red or starts flashing, stop and call us back for a breakdown service. For the appointment, may I have your name, please?
Caller: Ferrandi, Jean-Marc. Ferrandi, with two Fs, well, one F and one N... no, F-E-R-R-A-N-D-I. There you go.
Agent: Thank you, Mr Ferrandi, F-E-R-R-A-N-D-I. What is the vehicle's licence plate?
Caller: So, it's GH-482-KT. I'll spell it: G for George, H for Henry, 482, K for Kilo, T for Tango.
Agent: Let me repeat: G, H, 4, 8, 2, K, T, is that right?
Caller: Yes, that's it, GH-482-KT. There.
Agent: Perfect, thank you. What model is your vehicle and what is its approximate mileage?
Caller: It's a Peugeot 3008, 2019. It's done about 84,000 km, roughly.
Agent: Thank you, a 2019 Peugeot 3008 with about 84,000 kilometres, for an engine warning-light diagnosis and an oil change. What phone number can we reach you on?
Caller: It's 06 12 34 56 78. You can call me back on that, I'm often on the road, but I pick up.
Agent: Let me repeat: 06 12 34 56 78, is that right? Before Friday, I can offer you Thursday 12 November at 8:30, 10am, 2pm or 3:30pm. Which slot would suit you?
Caller: Yes, that's right. Well, Thursday doesn't really suit me, for me it's more Tuesday or Thursday morning. Do you have anything in the morning?
Agent: In the morning, I can offer you Thursday 12 November at 8:30 or 10am, or otherwise Tuesday 17 November at 8:30 or 10am, but that will be after Friday. Which of these slots would suit you?
Caller: Ah no, Tuesday the 17th is too late, I need it before Friday. All right, Thursday the 12th at 8:30 works for me.
Agent: That's booked, Mr Ferrandi: appointment on Thursday 12 November at 8:30 for the engine warning-light diagnosis and the oil change on your Peugeot 3008, at 140 avenue de la Libération. Is there anything else I can do for you?
Caller: Ah, perfect, thanks a lot. Say, before I hang up, could you tell me how much an oil change costs on a 3008?
Agent: Unfortunately I can't give you a price over the phone, but a quote will be drawn up at the workshop on Thursday before any work is done. If you'd rather know the price beforehand, I can ask an advisor to call you back within 24 hours. Would that suit you?
Caller: Oh, no price at all? Fine, I'll take the quote on Thursday then. No, no need to call me back, I'll be on the road. Thanks a lot, have a good day.
What would your agent do?
We run our business scenarios on your voice agent and walk you through the report.
Book a free pilotAudit by ProvaCall