Searched Generation (RAG) is rapidly emerging as a robust framework for organizations seeking to leverage the full power of AI generated using business data. Fills Rag Bridges General AI capabilities and domain-specific expertise as companies seek to leverage their own knowledge base beyond the general AI response.
Hundreds, perhaps thousands of companies are already using RAG AI services, and adoptions are accelerating as technology matures.
Also: I tested 10 AI content detectors and these 5 were correctly identified each time.
That’s good news. Bad news? According to AI researchers at Bloomberg, Rag can also significantly increase your chances of getting a dangerous answer.
Before you get into danger, check out the rags and their benefits.
What is a rug?
Rag is an AI architecture that combines the strength of generated AI models such as Openai’s GPT-4, Meta’s Llama 3, and Google’s Gemma with information from company records. Rather than relying solely on LLMS’ “pre-trained ‘world knowledge’, RAG can access and infer and infer databases, external knowledge stored in documents, and live in-house data streams.
When a user submits a query, the RAG system retrieves the most relevant information from the first curated knowledge base. Next, send this information to LLM along with the original query.
Maxime Vermeir, senior director of AI strategy at Abbyy, describes Rag as a system that can be generated not only from training data but also from specific, up-to-date knowledge.
Why use a rug?
The advantages of using RAG are clear. LLM is powerful, but does not have information specific to your business’s products, services, or plans. For example, if your company operates in a niche industry, your internal documents and unique knowledge are far more valuable to the answer than what you find in a public dataset.
By providing LLM with access to real business data (these PDFs, word documents, or FAQs), you can get more accurate and on-point answers to your questions.
Also: 60% of AI agents work in IT departments – this is what they do every day
Additionally, rags reduce hallucinations. This is done by grounding AI answers to trusted, external or internal data sources. When a user submits a query, the RAG system retrieves relevant information from the curated database or document. It provides this de facto context for the language model and generates responses based on both its training and the evidence obtained. This process makes AI less likely to create information. The answer is because you can trace back to your own internal sources.
“We use search engines to anchor the model’s response to case law, articles, or whatever is necessary, rather than answering based on memory encoded during the initial training of the model.”
Rag AI engines can still create hallucinations, but are less likely to occur.
Another advantage of RAG is that it allows you to extract useful information from long-standing, unorganized data sources that are otherwise difficult to access.
Previous rag problem
RAG offers a great advantage, but it’s not a magic bullet. With your data, the phrase “Um, bad,” comes to mind.
Related issues: If there is outdated data in the file, Rag will pull out this information and treat it as the truth of the gospel. It quickly leads to all kinds of headaches.
Also, do you need a Generated AI LLMS integrated with business data? I need a rag
Finally, AI is not smart enough to clean up all your data. Before lag-enabled LLM becomes productive, you need to organize your files, manage Rag’s vector database, and integrate them with LLM.
The danger of newly discovered rags
Here’s what Bloomberg AI researchers have found: Lugs can actually make the model “safe” and reduce the reliability of production volumes.
Bloomberg tested 11 major LLMs, including GPT-4O, Claude-3.5-Sonnet, and Llama-3-8 B, using over 5,000 harmful prompts. If LLM was rag-enabled, the model that rejected an insecure query in a standard (non-rag) setting produced a problematic response.
They found that even “safe” models increase dangerous power by 15-30% in RAG. Furthermore, longer search documents were correlated with higher risk, as LLMS struggled to prioritize safety. In particular, Bloomberg reported that even the very safe models “refused to answer almost all harmful queries in the rag setting, but become more vulnerable in the rag setting.”
Also: Why is it such a dangerous business to ignore AI ethics and how to do AI right?
What “problematic” results? Bloomberg, as you might imagine, was looking at the financial results. They saw AI leaking sensitive client data, creating misleading market analyses and generating biased investment advice.
Additionally, RAG-enabled models are more likely to generate dangerous answers that can be used in malware and political campaigns.
In short, as Amanda Stent, Director of AI Strategy and Research in Bloomberg’s CTO Office, said, “This counterintuitive finding has extensive implications given how it is used in GEN AI applications such as customer support agents and question-answer systems, to ensure that the output is appropriate.”
Also: Some secret AI companies can crush free society, researchers warn
“The unique design pull of Rag’s external data dynamically creates unpredictable attack surfaces. Mitigation requires layered safeguards, not only relying on the claims of model providers,” added Sebastian Galeman, responsible AI head at Bloomberg.
What can you do?
Bloomberg proposes creating a new classification system for domain-specific risks. Companies deploying RAG should improve their guardrails by combining business logic checks, fact-verification layer, and tests from the Red Team. For the financial sector, Bloomberg advises to look into and test RAG AIS for potential confidential disclosures, counterfactual narratives, fairness issues, and financial services fraud issues.
These issues must be taken seriously. As US and EU regulators step up AI scrutiny in finance, RAGs will be strong, yet demand strict domain-specific safety protocols. Finally, it’s easy to see companies sued if AI systems are not only poor to clients, but also offer completely wrong answers and advice.
Want more stories about AI? Sign up for our weekly newsletter, Innovation.
