Execution, Optimization, and Exploration: Evaluating Scientific AI Through RF Engineering

When we evaluate AI systems, we often focus on whether they can produce the correct answer. Done wrong, evals turn into “trivia,” instead of an ability to accomplish a complex task. But in science and engineering, the fact recall is rarely the most important question.

In my years at Purdue as a professor teaching electromagnetics and radio frequency (RF) engineering, I discovered that the most revealing problems were not those with clear solutions. Instead, they were the problems that forced students to determine what kind of thinking was required. Was the task simply to execute a known process even through ambiguity? Was it to optimize a design within established often conflicting constraints? Or was it to explore an unfamiliar space where the path forward was uncertain?

These three modes of problem solving, execution, optimization, and exploration, represent progressively more challenging forms of reasoning. A student who can execute a calculation demonstrates competence. A student who can optimize a design demonstrates judgment. A student who can successfully explore an unknown design space demonstrates creativity, adaptability, and scientific intuition.

The same framework can be used to evaluate modern AI systems. Rather than asking whether an AI can answer a question, we can ask a more meaningful question. Can it recognize what type of problem it is facing and apply the appropriate strategy? Can it follow established procedures when they exist, improve solutions when tradeoffs must be balanced, and explore new possibilities when no obvious answer is known? I map these loosely to the progression of students as they specialize in the field from their senior year specialization courses, through graduate classes, to PhDs exploring new topics.

To examine these questions, I revisited a series of RF engineering challenges similar to those I would give through my time teaching. These problems span the spectrum from well-defined calculations to open-ended design challenges that often require multiple rounds of experimentation, simulation, learning, and revision.

The results offer a useful perspective not only on how far AI has progressed but the value of Discovery for having the right human-machine interface to solve complex problems from both a scientific and engineering lens.

Senior Capstone Course Exam Question - Execution in ambiguous tasking:

Prompt 1: Design a waveguide that carries 3 modes from 5 to 7 GHz

One question that I pulled from an exam was the design of a waveguide with specified mode structure. This question, the design of a waveguide that carries three modes, is how I personally test each model as they are released to see how well they handle ambiguity. This nominally is an execution task, but with a few wrinkles. It requires the basic understanding of the concepts of waveguides, but also subtlety in how to interpret the question. In a typical homework question, you may ask to apply formulas from the textbook to ensure that the student knew of the variables and how to apply them effectively. The correct application of known formulas is the bare minimum for understanding the field. If this is interpreted as only a rectangular waveguide problem, then it is relatively simple application of known formulas, but has no answer. By leaving it vague, it allows the student to express the options before they chose the right calculations.

A human RF engineer immediately notices several ambiguities:

• Which three modes?
• TE only, TM only, or mixed?
• Are degeneracies acceptable?
• Is the objective three propagating modes simultaneously?
• Is mode purity important?
• What insertion loss is acceptable?
• What physical size constraints exist?
• Is manufacturability important?
• Is this an eigenmode problem or a driven-port problem?

A good engineer would either:

1. Ask clarifying questions, or
2. Make reasonable assumptions and document them.

That is actually the first intelligence test, and then execution begins. The essence of whether intelligence is being deployed in answering this question is whether the agents know when to clarify or when to execute, discerning when should the topic be further explored, or alternatively when do you commit given assumptions. These are all indicators of high level of reasoning and understanding of a topic relative to the simple deployment of known formulas.

Once it is clarified that there should be no new mode creation from 5 to 7 GHz, this becomes significantly trickier. This harder interpretation eliminates the classic and simplest design (a rectangular design) where there are clean analytical solutions. Since there are no answers for this line of reasoning, this eliminates hunting and pecking approaches of applying the analytical formulas where the model may iteratively just keep searching for the right dimensions. There are an infinite number of possible dimensions and none of them will work.

Using Discovery helps the user understand the various stages that will be executed by the agents and allow for human prodding or guidance as desired. This problem, even with the ambiguity and the lack of a solution with the primary approach, is now successfully managed if using the Discovery harness. The agents sort through the options, realize that the rectangular waveguide is a null solution, explore various options before settling on a ridge waveguide design. The model knows that there is not an analytical solution, so creates an eigenmode solver for the cross section. It iterates the design in this relatively quick simulation space before calling Ansys HFSS to confirm the results.

To solve this problem, the system autonomously accomplishes many tasks:

1. Discovery sorts through the approach to the problem, sorting through the questions that give rise to the ambiguity clarifying its approach it will take.
2. Analyzes the analytical formulas to realize that there is a null solution for standard waveguides
3. Realizes that a ridge waveguide is a viable alternative that could meet the criteria
4. Creates an eigenmode solver to quickly iterate through design choices for the ridge waveguide
5. Validates with a full wave solver that the dynamically created eigenmode solution is accurate
6. Creates the physical structure, assigns the boundary conditions, the port definitions, and the sweep parameters that solve Maxwell’s equations
7. Exports the S-parameters to validate the expected results applying the correct analysis to ensure its hypothesis was correct

The details of this reasoning and the design are in the report. The autogenerated writeup of its initial actions are useful to explore, which show the way the agents approached the problem without relying on full wave simulation. (report).

Figure 1 – Fully autonomously designed ridge waveguide that meets the desired three mode condition from 5 to 7 GHz.

Verdict: The Discovery system was able to exceed my expectations in the complexity of using the tools and managing the ambiguities while sharing the details needed to understand the scientific basis for its decisions. In the past the models have had a difficult time representing three-dimensional objects, but in this case it could not only draw them but assign the right boundary conditions to solve finite element solutions. This is a task that I would expect a top senior level student to have handled, and the Discovery framework allows for completion of this task with an understanding of why the subtasks were chosen.

Early graduate level design project: Design optimization of codependent variables.

Prompt 2: Design a horn antenna that has 15 dB absolute gain at 5 GHz

A slightly more difficult task is “Design a horn antenna that has 15 dB absolute gain at 5 GHz”. While this is somewhat straightforward from an antenna theory, the precise creation of the proper gain including reflection loss can be multifaceted. In this case the aperture size defines the gain, but the reflection loss is dependent on the taper since that can create reflection loss that hurts the absolute gain. So it is a balance to get the right aperture size and taper that will fit the final parameter. The essence of this question is whether the system can balance competing objectives and learn the underlying tradeoffs of the semantic based optimization.

The Discovery-based agentic process knew that the first steps were to build an analytical model that would give the rough estimate of the aperture of the horn antenna. The model then explored the workspace to realize that I had Ansys with HFSS within my boundary of access within my local computer and how to interface with it. This, in addition to organizing my notes from my Purdue lectures on the Discovery Bookshelf, was the knowledge gathering phase. It then used the analytical understanding of the relationship between the aperture size and the Fourier transform to the far field to predict the directivity. The agents then created the shape of a horn antenna with a suitable aperture size and automatically applied the correct boundary condition and simulation parameters. It extracts the S-parameters and the far field plots analyzing them to compare to the initial goals. Initially, the theory doesn’t account for the practicalities of the taper, so the realized gain was less than expected. While there is no formula that describes the angle of the antenna flare, the trends can be described. Too severe of a taper creates more reflection loss. The tradeoffs can be readily described but not easily mathematically represented. However, the Discovery loop performs semantic based optimization to learn the right approach to dialing in the result.

The initial analytical formulas of the gain are precise, but not account for phase taper and reflection loss. So when to trust the analytical formulas, when to follow the semantic design guidelines for taper, and when to make more severe adjustments like potentially adding a matching stub are all part of a trade space that can be described.

While specific to RF design, this is an example of intelligent semantic optimization using the tools of the trade to iterate to a final design. This occurs quite often in science and engineering when the role is to balance many variables that are in a tradeoff, often without a precision representation. The analytical formula creates an expected trend that the gain would increase with a larger aperture, but in simulation this trend was broken in one of the iterations (C3). The larger aperture caused more reflection loss than the increase in directivity, due to the severity of the flare from waveguide to aperture. The next round then tested its assumptions by making a smaller aperture which caused a smaller angular transition, and the system learned that this actually increased gain noting the impact of the gradual taper. The antenna had an increased gain because of the recovery of the reflection loss, and this lesson learned was stored for successive looping. Having recognized the core tradeoff, it made the next iterations with a much larger aperture, but with a more subtle angular flare to keep reflection loss low (C4 – C6). This initially overshot the goal, but in the last step (C7) it corrected the design to hit nearly 15 dB exactly.

The process took 7 iterations which I would broadly classify as learning the optimization trade space (C1 – C4), then using that knowledge hone in on the design (C5-C7).

Figure 2 – Example of the learning cycle of the iterative design to understand the trade space of the horn antenna features.

Verdict: Ultimately the ability for the agents to execute the full design cycle was beyond what I would have expected going in. The execution agents have to know multiple subtleties of the HFSS software package like the port must be flush with the radiation boundary, and/or how to define metals within the air cavity to not have overlapping material definitions. Earlier examples failed in the three-dimensional construction of the simulation. These are typical errors that new users encounter when using a new tool with so many parameters to adjust. You not only know how to create the 3D object, you have to understand the creation in the construct of the simulation package and the applications of the differential equation boundary conditions. The 3D object drawn in this scenario has some complexity, and while I would typically think of a continuous flare it chose to draw the object in discrete sections, showing a bit of a limitation in 3D object representation. This is not a typical shape of a horn antenna and feedback from the manufacturing process likely asks for a more contiguous design. However this still meets the design goals.

This total execution was autonomously run in less than an hour running through 7 different designs, extracting the S-parameters and the radiation parameters and learning the lessons from each run. While I think I could have come up with a similar design, it would have taken me much longer.

Qualifying Exam Level: Unbounded Exploration

Prompt 3: Design and attach a horn antenna that has maximum radiation at 0 degrees for the first mode, 30 degrees for the second mode, and 10 degrees for the third mode

I then gave Discovery a much harder problem that I wouldn’t expect my students to know the answer to. In fact, I wasn’t certain that there is an answer, but that is not the important aspect. This would be the type of question I would give in a one-on-one qualifying exam where you are probing the students ability to think through a complex problem, much more than you are looking for the right answer. The essence of this question is whether you can intelligently search for design strategies in a relatively unbounded exploration space.

In this task I am asking to attach a horn antenna to the end of the waveguide with 3 modes. Each mode is then expected to have a peak of radiation at 0, 30 and 10 degrees respectively. These are in direct tension with each other and the designer needs to figure out how to alter the phase distribution across the aperture, or otherwise know how to isolate the three modes. A dielectric insertion can slow the propagation of one of the modes to skew its radiation, but will impact the other modes simultaneously.

By utilizing Discovery with the CLIO project harness (CLIO link), the system explored various options to try to handle the highly competing tradeoffs. It correctly decided what could be done to understand the problem by quickly working through a rectangular aperture. This use of a “quick and dirty” or “back of the envelope calculation” to understand the fundamental tradeoffs shows a layer of intelligence. This allowed the system to calibrate when to use the analytical approach and to create multiple belief states, i.e. approaches to the design problem, that can be compared as they execute, learning from each other. It then called Ansys HFSS to confirm the design approaches.

Figure 4 – Analytical solutions for rectangular apertures to understand the trade space for design and why the desired parameters are in tension. This was the first analysis attempted on the path to exploration.

The system also used the analytical overview to recognize that even if the tension in the design can be met, the horn will have beam squint and peak movement across the bandwidth. Any design is likely to be narrowband at best. So a final solution needs to both hunt for maximum radiation angles and stability across frequencies from 5 to 7 GHz. Therefore the solution is fundamentally difficult which is correctly acknowledged.

It suggested multiple options to first explore a single aperture approach, all which were rational. These tried to execute the options learning from multiple concepts to try to balance the underlying tension, mainly focusing on ways to shift the phase of the third mode across the aperture while maintaining the first mode balance. These changes weren’t subtle tweaks in dimensions, but explorations of the types of designs that might get to convergence, such as leaky slot waveguides, asymmetric dielectric loading and lenses, inclusion of septum and metallic fins.

Most importantly for me, it gave the assumptions and regrets of the various approaches at critical decision points, so that I could understand the complexities of the decisions it needed to make. These explorations ran for over 2 days in total, and therefore reading the agentic execution logs is difficult at best. However, the system appropriately flagged where difficult decisions were made, when it made semantic leaps, detailing the assumptions at each critical stage and the regrets of the choices it had to make, allowing a human to gauge where to potentially follow further or course correct.

Figure 5 – Some of the many belief states that the designs iterated through showing progress towards intelligent designs even in a hard set of boundary conditions.

 

Realizing that a single aperture would be very difficult to balance, especially across the bandwidth, the CLIO engine then decided to move to a different strategy. Instead of a single aperture, which admittedly was my intended approach to the question, it decided it was easier to split the modes into three separate waveguides. Then feed this split modes into three separate horns which the agents can shape in any direction. This is a simplification of the problem in many ways but complicates the mode separation. My first response is that this is “cheating” in that it simplifies the problem that I had envisioned yet hadn’t articulated. The final design that it comes up with is definitely compliant with the question. This is one structure but now separated into three apertures, each responsible for each mode. The hard part is getting the three way mode splitting to be done so that it feeds each waveguide feed to the horns independently. At this task did this tremendously.

Figure 6 – Cross Section showing the TE2,0 mode converting to the one of the three apertures, in this case the one that is bent to 30 degrees. All three modes were correctly steered to the right horn.

Verdict: While a single aperture design never closed, it did solve for a structure that meets the criteria. In fact, I learned alongside the exploration of different ways to approach this problem. The final result was different than I had envisioned, and I even learned a few tricks watching it solve the structure, like how it approached mode separation. The shapes were not random, and were strategically placed based on root principal physics, then verified.

A full solution was never my intention. More importantly, Discovery did show intelligent exploration based on the underlying physics and mode structure, flagging when critical decisions were made. The system further drew very complex shapes and simulated them with proper design philosophies and sound judgement. Hundreds of HFSS designs were iterated on, autonomously generated, analyzed with the lessons appropriately captured. Large “semantic leaps” in the approach were made when it was clear that the tension in the current design space was too much or not closing. The proper expression of assumptions, basis for the decision and the potential regrets when it had to make a hard choice.

Discovery allows for two required elements here. One, AI needs to share the critical decisions and the doubts of the decision making, so that a human can follow numerous iterations. And second, the full exploration needs the ability for guidance from the human. In very difficult designs, which of the criteria are you willing to relax? In this particular case, what is an acceptable beam squint across the bandwidth? Is mode isolation more important than beam squint? Did it correctly state the right assumptions at some of the critical design choices, or is there more to the problem? A human could interject on breadth versus depth, and guide how far to iterate to a given belief state and chase an idea as opposed to explore. These are all parts of the joint human agent decisions to make that would be needed for a realistic design that a human would sign up to represent. The ability to suggest those decision parameters and clearly expose the decision making for human feedback is essential to make this human directed exploration.

Impact on RF Education and Education in General

In this example I walked through how students at various stages will be able to use the Discovery agentic frameworks to harness models and solve different types of engineering problems. While we built Discovery for science, the concepts apply to engineering and design problems just as well. The implications for engineering education are significant, and of course these results are a double-edged sword for learning the foundational concepts. While the ability to walk through advanced designs quickly has potential to speed learning cycles, it can also eliminate those same cycles for those who chose to ignore the details blindly accepting the results. The desire to use this as a crutch to get an answer quickly will be alluring.

A good educator can use these examples to provide realistic, hands-on experience quickly, complementing laboratory work that puts theory into practice. By my estimate, the first problem would have taken me an hour, the second at least a day, and the last a month to explore as many designs as the system did. Simply creating hundreds of HFSS models would be time-consuming and add little to understanding the underlying physics. With Discovery, these problems can be addressed in minutes to days, depending on their complexity, while the user remains an active design partner. I learned substantially from the process; for example, the approach used to separate the three modes and feed each waveguide was new to me.

I have told my team “How you use a map app when you land in a new city” is a good test of whether you will use AI correctly for learning. Do you use the purple line to arrive at your destination faster, or do you use the fact that you have an Atlas in your pocket to explore the features of the city. Most technology advances can be used for either shortcuts or furthering our understanding, and this will be an extreme test of that choice. These capabilities will even further the need for a foundation of curiosity, which needs to be taught at the most basic levels.

For classrooms, the question is how to foster the learning environment to ask the follow-on questions and to use the speed of iteration to the advantage of learning cycles. Instead of describing invisible waves and only showing mathematical formulas or worse describing how an antenna or chip could be designed without ever having to do it, you can have a student partner through a complete design. In my own area of specialization, electromagnetics has been the domain for description of invisible waves described by obscure formulas. Only certain types of learners can compete in the top classes. Visual or experiential learners have been traditionally disadvantaged compared to formulaic learners. This can bring rapid and visual experiences into the curriculum, expanding the appeal to certain types of students.

But the question will be if the tool is used as a crutch or to enhance learning. Used on its own by students, I expect there will be even further separation from those that are truly curious or just expedient. The Khan academy has shown a path at lower levels, but only when the AI tools are used by good teachers and a system where they know how to use the tools to ask the right questions ((99+) Credit the Educators, Not Just the Technology | LinkedIn). In my estimation undergraduate and graduate faculty are actually very good at knowing how to probe a student of whether they really learned from the experience or just regurgitating. Tools like Discovery won’t be a successful tool for teaching on their own but will put a premium on the great educator that motivates the need for understanding. The need for great scientists and great educators will put a severe demand on human skills to use this correctly. These tools won’t be nearly as valuable for generalists, and even understanding what is going on takes skill. For education, I foresee a separation in good educators and those who use the university as a crutch for research positions, warranting a rethinking of the incentives of the system that traditionally has rewarded research papers which are highly likely to change as the metric of valuable output.