The elders all say that this kind of thing—chatting and bragging—is really fun.
The author shares his experience of consulting a Shopify-developed documentation assistant powered by a large language model (LLM) when upgrading Shopify email notification templates. He aimed to detect if an order contained items shipped through Shopify Collective, and the assistant quickly provided code to check if "Shopify Collective" existed in order.tags. However, the code failed in practice because the tag wasn't added at the time of notification generation, only being added later through a mysterious Shopify process. Further testing revealed that the "Shopify Collective" tag didn't exist when confirmation emails were sent, making the assistant's answer not only ineffective but also exposing a flaw: the assistant had guessed based on default documentation without considering the timing issues in actual workflows. The author questions whether frequent such arbitrary responses from documentation assistants could lead to higher error costs than the occasional benefit of accurate information. Many developers in the discussion shared similar concerns, arguing that AI assistants' intuitive answers could lead to unstable guidance for users. Some pointed out that incorrect documentation responses are more frustrating than no response at all, even comparing it to salespeople providing irrelevant and impractical answers to technical issues. Some responders emphasized that official documentation built through accumulated experience is more reliable for precise operations. Additionally, the discussion mentioned that even with systems like RAG (Retrieval-Augmented Generation) to assist queries, balancing rapid responses with stability and accuracy remains challenging. Several speakers cited other features (e.g., batch adding discount codes) as examples where AI answers might vary unpredictably due to insufficient contextual understanding. Overall, the consensus was that technical documentation should be based on real testing and careful writing, rather than relying on AI assistants that occasionally produce fantastical answers.
https://news.ycombinator.com/item?id=44507244