Unable to generate detailed or multi agent report with smaller language model

[sjans1]

Smaller language models such as llama-3.1-8b-instruct and gemma-2-2b-it work for creating summary research report, but cause the researcher to get stuck when creating a detailed or multi agent report. I understand that the framework really depends on the power of the large language model, but these problems could be caused because of the inaccuracy or unnecessary complexity of the prompts.

Specifically there are two areas where it can get stuck:

• generating subtopics
• multi agent responses that request JSON (editor, reviser, writer)

Subtopics
Generating subtopics relies on the PydanticOutputParser and this results in the prompt including this very complicated bit of instruction for the JSON format:

The output should be formatted as a JSON instance that conforms to the JSON schema below.
As an example, for the schema {"properties": {"foo": {"title": "Foo", "description": "a list of strings", "type": "array", "items": {"type": "string"}}}, "required": ["foo"]}
the object {"foo": ["bar", "baz"]} is a well-formatted instance of the schema. The object {"properties": {"foo": ["bar", "baz"]}} is not well-formatted.

Here is the output schema:

{"$defs": {"Subtopic": {"properties": {"task": {"description": "Task name", "minLength": 1, "title": "Task", "type": "string"}}, "required": ["task"], "title": "Subtopic", "type": "object"}}, "properties": {"subtopics": {"default": [], "items": {"$ref": "#/$defs/Subtopic"}, "title": "Subtopics", "type": "array"}}}

This often is too complicated for the smaller language model. Maybe this should be simplified to what is really needed in the same style as when generating the agent or sub queries or editor, reviser and writer response JSON, or have a fallback to a more simple prompt for generating subtopics.

I already get good results when I replace that part of the prompt with:

The output must be a JSON object like this:{ "subtopics": [{"task": "example topic 1"}, {"task": "example topic 2"}, {"task": "example topic 3"}]

Multi agent planning and writing
The multi agent report depends on JSON responses for the editor, reviser and writer. Sometimes the LLM simply does not follow the instructions, perhaps there should be some attempt to repair the JSON similar to what is done when generating the search queries?

[sjans1]

I believe that similar issues have been reported previously by others using local language models that are less powerful. For example:

• Crashes instead of displaying "Could not find any answers for this topic" when it wasn't able to find any relavant information. #605 is about sub topics or sub queries and JSON parsing
• llama3 keyerror 'server' #701 is about creating agents and JSON parsing

[assafelovic]

Hey, these are limitations of using other models. GPTR is heavily tested on OpenAI models. Any other models are at your responsibility. In either case we'd love to learn from your experience so thank you for raising this feedback!