RAG-Critic: Leveraging Automated Critic-Guided Agentic Workflow for Retrieval Augmented Generation

rag-critic

Guanting Dong, Jiajie Jin, Xiaoxi Li, Yutao Zhu, Zhicheng Dou ^✉; Ji-rong Wen

Gaoling School of Artificial Intelligence, Renmin University of China.
^✉ Corresponding Author

The official model weight for RAG Error-Critic Model (3B)， which used to identify the error response of RAG models. Here we illustrate the input-output format for our model:

Example1:

RAG Input1:

You are a critical system designed to provide useful error type tags for retrieval-augmented generation (RAG) tasks. Your goal is to assist in detailed error analysis to improve the performance of AI assistants. Below are the [Question], the top-5 retrieved relevant [Passages], and the [Model's Prediction] for the RAG tasks.

                Question: Who was born later, Serafino Morazzone or Alice Rachele Arlanch?
                Passage 1: "Madame Pasca"
Madame Pasca Alice Marie Angèle Pasquier (November 16, 1833 – May 25, 1914), better known by her stage name Madame Pasca and also known as Alix-Marie Pasquier, was a French stage actress. Pasca was born on November 16, 1833 in Lyon to Jeanne-Marie-Antoinette-Eugénie Morin and Louis-Joseph Séon, a merchant. She took singing lessons from François Delsarte, but had limited vocal range and was encouraged by Delsarte to pursue acting instead. She married M.Pasquier in 1855 and when a widow she first appeared as an actress at the ""Gymnase"" in Paris in 1864. As an actress, Pasca received significant critical attention.
                Passage 2: "Alice Rahon"
Alice Rahon Alice Phillipot (Alice Rahon) (8 June 1904 – September 1987) was a French/Mexican poet and artist, whose work contributed to the beginning of abstract expression in Mexico. She began as a surrealist poet in Europe, but began painting in Mexico. She was a prolific artist from the late 1940s to the 1960s, exhibiting frequently in Mexico and the United States, with a wide circle of friends in these two countries. Her work remained tied to surrealism, but was also innovative including abstract elements and the use of techniques such as sgraffito and the use of sand for texture.
                Passage 3: "Serafino Morazzone"
Serafino Morazzone Blessed Serafino Morazzone (1 February 1747 - 13 April 1822) was an Italian Roman Catholic priest. Morazzone served as a simple parish priest in Lecco from his ordination as a priest in 1773 until his death and became noted amongst the faithful for his personal holiness and dedication to the sacraments. Morazzone's beatification process started in 1864 but later halted due to issues in Milan and the cause remained inactive until its revitalization in the 1950s. He was named as Venerable in 2007 and was later beatified at the Milanese Duomo on 26 June 2011. Serafino Morazzone was
                Passage 4: "Mario Serandrei"
Mario Serandrei Mario Serandrei (23 May 1907 – 17 April 1966) was an Italian film editor and screenwriter. Born in Naples, he started in the film industry in 1931 as an assistant director. He edited over two hundred films during his career, and worked steadily until his death in 1966. As an editor, Serandrei's credits included Federico Fellini's ""Il bidone"" (1955), Pietro Francisci's ""Hercules"" (1958) and ""Hercules Unchained"" (1959), Valerio Zurlini's ""Estate violenta"" (1959), ""La ragazza con la valigia"" (1961) and ""Cronaca familiare"" (1962), and the Robert Aldrich/Sergio Leone film of ""Sodom and Gomorrah"" (1963). He enjoyed a long working
                Passage 5: "Serafino Morazzone"
miracle on 2 April 2011 and confirmed the beatification. Cardinal Angelo Amato presided over the beatification on 26 June 2011 outside of the Milanese Duomo. The current postulator for this cause is Dr. Francesca Consolini. Serafino Morazzone Blessed Serafino Morazzone (1 February 1747 - 13 April 1822) was an Italian Roman Catholic priest. Morazzone served as a simple parish priest in Lecco from his ordination as a priest in 1773 until his death and became noted amongst the faithful for his personal holiness and dedication to the sacraments. Morazzone's beatification process started in 1864 but later halted due to issues
                Model's Prediction: To answer the question of who was born later, Serafino Morazzone or Alice Rachele Arlanch (Alice Rahon), we need to compare their birth dates.

1. First, we find the birth dates of both individuals from the given passages:
   - Serafino Morazzone: 1 February 1747
   - Alice Rachele Arlanch (Alice Rahon): 8 June 1904

2. Next, we compare the two dates:
   - Serafino Morazzone was born in 1747, which is significantly earlier than 1904, the year Alice Rachele Arlanch was born.

Critic Output1:

 {'Judgement': 'Error', 'Error_analysis': "The model's prediction is incorrect because it concludes that Serafino Morazzone was born before Alice Rachele Arlanch (Alice Rahon), which is true, but the question asks who was born later. The correct answer should be Alice Rachele Arlanch (Alice Rahon), as she was born on 8 June 1904, which is later than Serafino Morazzone's birth date of 1 February 1747.", 'tag1': ['Irrelevant or Off-Topic Response', 'Irrelevant Information'], 'tag2': ['Off-Topic and Redundant Responses', 'Irrelevant Information Retrieval'], 'tag3': ['Logical Error', 'Misinterpretation of Question', 'Incorrect Fact Selection']}

Example2:

RAG Input2:

You are a critical system designed to provide useful error type tags for retrieval-augmented generation (RAG) tasks. Your goal is to assist in detailed error analysis to improve the performance of AI assistants. Below are the [Question], the top-5 retrieved relevant [Passages], and the [Model's Prediction] for the RAG tasks.

                Question: How far does the earth travel in a second?
                Passage 1: Second
sound travels about 343 meters in one second in air; light takes 1.3 seconds to reach Earth from the surface of the Moon, a distance of 384,400 kilometers. A second is part of other units, such as frequency measured in hertz (inverse seconds or second), speed (meters per second) and acceleration (meters per second squared). The metric system unit becquerel, a measure of radioactive decay, is measured in inverse seconds. The meter is defined in terms of the speed of light and the second; definitions of the metric base units ampere and candela also depend on the second. Of the
                Passage 2: Light-second
Light-second The light-second is a unit of length useful in astronomy, telecommunications and relativistic physics. It is defined as the distance that light travels in free space in one second, and is equal to exactly metres by the definition of metre (just over miles). Just as the second forms the basis for other units of time, the light-second can form the basis for other units of length, ranging from the light-nanosecond (299.8 mm or just under one international foot) to the light-minute, light-hour and light-day, which are sometimes used in popular science publications. The more commonly used light-year is also
                Passage 3: Light-second
Light-second The light-second is a unit of length useful in astronomy, telecommunications and relativistic physics. It is defined as the distance that light travels in free space in one second, and is equal to exactly metres by the definition of metre (just over miles). Just as the second forms the basis for other units of time, the light-second can form the basis for other units of length, ranging from the light-nanosecond (299.8 mm or just under one international foot) to the light-minute, light-hour and light-day, which are sometimes used in popular science publications. The more commonly used light-year is also
                Passage 4: Second
second per second for acceleration, and per second for frequency. Although the historical definition of the unit was based on this division of the Earth's rotation cycle, the formal definition in the International System of Units (SI) is a much steadier timekeeper: 1 second is defined to be exactly ""the duration of 9,192,631,770 periods of the radiation corresponding to the transition between the two hyperfine levels of the ground state of the caesium-133 atom"" (at a temperature of 0 K). Because the Earth's rotation varies and is also slowing ever so slightly, a leap second is periodically added to clock
                Passage 5: "Julian day"
the Sun and the targeted object, and for the second, the Earth is on the opposite side of the Sun from that object. Then, the two measurements would differ by about 1000 light-seconds: For the first measurement, the Earth is roughly 500 light seconds closer to the target than the Sun, and roughly 500 light seconds further from the target astronomical object than the Sun for the second measure. An error of about 1000 light-seconds is over 1% of a light-day, which can be a significant error when measuring temporal phenomena for short period astronomical objects over long time intervals.
                Model's Prediction: To answer the question of how far the Earth travels in a second, we need to understand that the distance traveled by an object is calculated by multiplying its speed by the time it travels. In this case, we know the speed of light (which is the speed of the Earth in a vacuum, as the Earth's movement through the air is negligible compared to the speed of light) and the time (one second).

1. First, we need to find the speed of the Earth in meters per second. Since the speed of light is 343 meters per second in air, and we are looking for the Earth's speed in a vacuum, we can use the speed of light in a vacuum, which is approximately 299,792,458 meters per second.

2. Next, we need to find the distance the Earth travels in a second in a vacuum. To do this, we multiply the speed of the Earth in a vacuum by the time (one second).

   Distance = Speed * Time
   Distance = 299,792,458 m/s * 1 s
   Distance = 299,792,458 meters

Critic Output2:

 {'Judgement': 'Error', 'Error_analysis': "The model's prediction is incorrect because it uses the speed of light in a vacuum (299,792,458 m/s) instead of the Earth's orbital speed around the Sun. The correct answer, as provided in the golden answer, is that the Earth travels approximately 29.78 km/s in its orbit around the Sun. The model's calculation is off by a factor of about 10,000, which is significant in the context of the question. The model also incorrectly assumes that the Earth's speed in a vacuum is the same as the speed of light, which is not accurate.", 'tag1': ['Incomplete Information', 'Inaccurate or Misunderstood Response', 'Irrelevant Information'], 'tag2': ['Insufficient or Incomplete Information Retrieval', 'Factual Inaccuracies', 'Irrelevant Information Retrieval', 'Misinterpretation of Queries and Information'], 'tag3': ['Incorrect Assumption', 'Omission of Key Information', 'Misalignment with Golden Answer', 'Factual Error']}

dongguanting
/

RAG-Critic-3B